Skip to content

feat(action-plugin): run dynamic-run over RPC - #16256

Draft
rgrinberg wants to merge 6 commits into
ocaml:mainfrom
rgrinberg:push-xzxwkzrvqrtl
Draft

feat(action-plugin): run dynamic-run over RPC#16256
rgrinberg wants to merge 6 commits into
ocaml:mainfrom
rgrinberg:push-xzxwkzrvqrtl

Conversation

@rgrinberg

Copy link
Copy Markdown
Member

Replace the temporary-file handshake used by dynamic-run with Dune RPC requests. Keeping the plugin alive lets staged computations preserve process state and avoids re-executing earlier stages while Dune builds newly discovered dependencies.

  • Record RPC-served dynamic dependency facts in the existing action result.
  • Preserve cycle detection when dependencies are built from the RPC handler fiber.
  • Use unpredictable action IDs bound to a single RPC session.
  • Keep the DAP client monad-parametric while the built-in blocking client uses the shared thread pool.

Move the reusable thread-pool implementation out of dune-scheduler so
lower-level libraries can use it without depending on the scheduler.
Parameterize worker creation so the scheduler retains Thread0's signal masking
and tracing behavior while other clients can use ordinary system threads.

Keep the existing worker-accounting and capacity tests on the scheduler
wrapper.

Signed-off-by: Rudi Grinberg <me@rgrinberg.com>
Replace the temporary-file dynamic-run handshake with versioned DAP RPC
procedures. Dune registers each active action, passes its RPC endpoint and
action ID to the plugin, and serves dependency requests while staged execution
continues in one process.

Provide a monad-parametric DAP client with explicit structured errors, plus a
blocking implementation backed by the shared thread pool. Keep the DAP payload
representations private while exposing the request witnesses required by
caller-provided Dune RPC clients.

Record RPC-built dependencies in the action result and remove the obsolete
action-execution continuation used to respawn plugins between stages. Update
tests, procedure digests, and documentation for single-process staging.

Signed-off-by: Rudi Grinberg <me@rgrinberg.com>
Extend the action-plugin helper so a rule producing one target requests a
second target whose rule depends on the first. Record the existing timeout to
demonstrate that serving the dependency from an unrelated RPC fiber loses the
Memo cycle-detection path and deadlocks.

Signed-off-by: Rudi Grinberg <me@rgrinberg.com>
Capture the Memo cycle-detection context before action execution and restore it
when an RPC handler builds dynamic dependencies. This reconnects the handler's
build to the active rule's Memo stack, allowing indirect cycles to be detected
before either side blocks.

Use the general Memo cycle detector for direct target dependencies as well,
removing the duplicate target matching logic and restoring Dune's standard
cycle diagnostics.

Signed-off-by: Rudi Grinberg <me@rgrinberg.com>
Extend the blocking action-plugin helper to hold an initialized action open and
publish its action ID and RPC endpoint. Start a second blocking client with the
same ID to demonstrate that the server currently accepts a live action from an
unrelated RPC session.

Signed-off-by: Rudi Grinberg <me@rgrinberg.com>
Generate unpredictable action IDs instead of process-local counters and bind an
action to the RPC session that initializes it. Reject duplicate initialization,
uninitialized dependency requests, and requests from any other session.

Update the blocking-client regression to verify that a second session can no
longer reuse a live action ID.

Signed-off-by: Rudi Grinberg <me@rgrinberg.com>
@rgrinberg rgrinberg changed the title refactor(action-plugin): run dynamic-run over RPC feat(action-plugin): run dynamic-run over RPC Aug 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant